Jacobian-Free Three-Level Trust Region Method for Nonlinear Least Squares Problems

Nonlinear least squares (NLS) problems arise in many applications. The common solvers require to compute and store the corresponding Jacobian matrix explicitly, which is too expensive for large problems. In this paper, we propose an effective Jacobian free method especially for large NLS problems because of the novel combination of using automatic differentiation for J(x)v and J (x)v along with the preconditioning ideas that do not require forming the Jacobian matrix J(x) explicitly. Together they yield a new and effective three-level iterative approach. In the outer level, the dogleg/trust region method is employed to solve the NLS problem. At each iteration of the dogleg method, we adopt the iterative linear least squares (LLS) solvers, CGLS or BA-GMRES method, to solve the LLS problem generated at each step of the dogleg method as the middle iteration. In order to accelerate the convergence of the iterative LLS solver, we propose an inner iteration preconditioner based on the weighted Jacobi method. Compared to the common dogleg solver and truncated Newton method, our proposed three level method need not compute the gradient or Jacobian matrix explicitly, and is efficient in computational complexity and memory storage. Furthermore, our method does not rely on the sparsity or structure pattern of the Jacobian, gradient or Hessian matrix. Thus, it can be applied to solve any large general NLS problem. Numerical experiments show that our proposed method is much superior to the common trust region method and truncated Newton method.